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SUPPORTED BY THE NATIONAL SCIENCE FOUNDATION 



Wayne W, Welch and Victor L, WUlson 
University cf Minnesota 

I^. ^ Introduction 

During the past two years, the National Science Foundation (NSF) has 
supported a program of implementation activities under the auspices of the 
Instructional Improvement Implementation Section, The purpose of this new 
program Is to provide assistance "to those who are concerned with the 
implementation of science curricula and Improved Instruction within their 
own school systems and classrooms/'^ NSF supports projects, ordinarily 
at colleges and universities, designed to Implement major curricula at the 
precollege level In science, mathematics, and social sciences. Several 
different mechanisms are used to accomplish the desired curriculum changes. 
These mechanisms, called delivery systems here, provide for Interactions 
between schools and colleges and were designed to familiarize teachers 
with new curricula, 

II . Problem 

An evaluation of five different Implementation projects was carried out 
during 1973-74, Each project used a different delivery system to achieve 
Its Implementation objectives. The purpose of the evaluation was threefold: 

(1) to develop appropriate methodology for evaluating implementation projects, 

(2) to characterize several selected delivery systems, and (3) to learn more 
about the process of dissemination and utilization of science curricula. 
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We wish to express our appreciation to the Project Directors who willingly 
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III * Procedure 

A model of knowledge dissemination and utilization recently proposed by 
2 

Havelock provides a conceptual framework to evaluate the various delivery 
systems. This model portrays the implementation process as an interaction 
or linkage between a potential "user" and a potential "resource." The purpose 
of the linkage is a transfer of knowledge. It can be analyzed into five 
categories by the question: " Who says what to whom by what channel to what 
effect?" This question is applied to the current problem in Figure 1, 



Figure 1 

Knowledge Transfer in NSF Implementation Program 
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It is noted in Figure 1 that the "channel" for implementation varies 
from project to project. In one instance, it is a two-week summer workshop 
at a local college; in another it is a fifteen-week in-service program in a 
local district. The various "channels"~we call them delivery systems— are 
the prime focus of this study. 

Havelock has found that seven general ' factors appear to account for or 
are related to effective knowledge dissemination and utilization.*^ By applying 
these seven f actors^Ccriteria) to the delivery systems, a clearer picture of 
the strengths and limitations of the systems should be available. 



ERIC 



4 



3 



Five different delivery systems were the focus of this study. All 
projects were funded by NSF and are typical of the Implementation program 
started by the Foundation in 1973 • A series of on-site visits, interviews, 
observations, and questionnaires were used to describe the major character- 
istics of each delivery system. These characteristics were summarized in 
..a series of reports prepared for projects directors and NSF. 

A panel of nine judges read each report and gained familiarity with each 
project through a series of staff discussions which occurred periodically 
throughout the life of the evaluation. These judges were then asked to rate 
each delivery system against the seven factors proposed by Havelock, et.al. 
A two-way analysis of variance with repeated measures was computed to 
determine if significant differences existed among projects and to determine 
the degree to which the. projects, as a group, satisfied the seven qualities 
for successful implementation. Finally, judges' ratings were compared with 
several other descriptors of the several Implementation modes. ^ 

IV . Results 

A. Descriptions of the Delivery Systems ^ 

The five delivery systems are examples of what Havelock, et.al., 
term "temporary systems for promoting change." These systems are 
differentiated by six characteristics: users, location, time of 
instruction, instructors, instructional responsibility, and school 
district support. 

What was communicated in the five systems is not essential to 
differentiating them, so that the curricula studied need not be 
discussed A common reward was provided in all five systems. 
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Teachers who participated received graduate level credits from a 
university associated with the system. In most cases, teachers also 
received tuition support and some travel allowance. 

The first delivery system to be discussed is the Portal School. 
Instruction was given to groups of elementary or secondary school 
teachers, usually from the same school or district. The classes met 
weekly after teaching hours at a local school for approximately one 
semester. Instructors were themselves elementary or secondary teachers 
who had attended training sessions at the sponsoring university and 
had been certified. School districts supported the Portal School by 
providing meeting sites, some materials, and in some cases tuition 
costs, which were otherwise borne by teachers. Districts made 
commitments with the sponsoring university to implement the curriculum 
studies. 

The Accessible School delivery system was of a semester's duration 
with meetings once a week in the evening at a local junior college. 
Classes were taught by junior college instructors who had previously 
attended short training sessions at the sponsoring university. The 
teachers taught at local junior high schools. V,Hpwever, there was no 
district level participation. 

The Off-Campus Center delivery system provided semester-long 
instruction for secondary teachers. Classes met weekly at night in 
a local school that was made available by a cooperative school district. 
Instructors were themselves secondary teachers who completed a master's 
level program at the sponsoring university. Teachers' school districts 
allowed the use in schools of curricular materials developed or discussed 
at the Off-Campus Centers, but the districts provided no financial support 
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Another delivery system Is represented by the Sumner Workshops 
conducted at a sponsoring university. Each workshop presented one 
curriculum, and each lasted two weeks with eight hours of Instruction 
and laboratory work per day. Instructors were university faculty, 
and In several workshops secondary teachers were hired as assistant 
Instructors. No school district commitment was made concerning 
curriculum Implementation. 

The final delivery system to be discussed Is a cooperative type 
workshop. One or two teachers in several districts were trained at 
the sponsoring university as resources In the curriculum to be Imple- 
mented. Local workshops for teachers were conducted by Instructors, 
expert In the curriculum prior to the beginning of the school year. 
During the year periodic classes were held for teachers, In which 
university personnel participated. School districts which participated 
agreed to Implement the curriculum. 

B. Havelock's Factors 

Based on a review of nearly 4,000 studies of the dissemination 
and utilization process (D&U) Havelock and his associates have proposed 
seven general factors which account for most D&U phenomena. In Table 1 
a brief description Is provided of each factor as It relates to the 
channel through which knowledge is communicated. 

[Insert Table 1 here] 
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Table 1 

Brief Description of General D&U Factors^ 
Specific to Channel 



1 . Linkage 

2. Structure 



- allows direct contact; two-way interaction. 



systematic strategy; timing to fit user's 
problem-solving cycle. 



3. Openness 



flexible strategies; best channel allows informal 
communications between sender and receiver about 
the innovation. 



4. Capacity 



capacity of channel to carry maximum information; 
accessibility to maximum number of users in 
minimum time. 



5 . Reward 



channel which can convey feedback (+ and - 
reinforcement); most effective channel has 
best reward history for sender and receiver. 



6. Proximity 

7 . Synergy 



- easily accessible channel, familiar to the user. 



the number and diversity, continuity and 
persistence of different channels used to 
transmit the message. 
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C. Rating Scales 

A five-point rating scale was used to rate each delivery system on 

each of the seven general factors. The anchor descriptions for each 

point are listed below, 

5 - uses to fullest extent possible 

4 - system permits substantial amount 

3 - typifies somewhat 

2 - little more than chance 

1 - creates a negative situation 

Cell means for the nine judges were computed to determine the fit of 
each medium with each of the seven D&U factors. Although these means are 
potentially useful in a formative sense to project directors, they are 
not crucial for this paper and are not reported here. 

Row and column mean ratings were computed and are presented in Table 2. 
Both factor and^ delivery system are ranked in descending order. 

[Insert Table 2 here] 

The results of an analysis of variance^ of factor and delivery system 
differences is pres.ented in Table 3. Both main effects are significant 
at the p<.01 level, while the interaction effect is not significant. 
Apparently, differences do exist among projects in the extent to which 
they meet the factors taken as a group vary significantly in their 
characterization by the Havelock factors. They are rated relatively 
high on proximity, capacity, and structure, but rather low on openness 
and synergy. 

[Insert Table 3 here] 
An inter-rater reliability coefficient was determined using the 
intraclass correlation procedure proposed by Guilford.^ A consistency 
estimate of .62 was obtained. 
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Table 2 

Mean Ratings 
D&U Factor by Delivery System 



Portal Collaborative Centers Accessible Workshop 

Proximity 3.51 

Capacity 3. 31 

Structure 3.29 

Linkage 2. 18 

Reward 3 ^ Qy 

Openness 2.82 

Synergy 2.78 

3.54 3.43 3.02 2.87 2.82 3.14 
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Table 3 

Analysis of Variance 
Single Group Two-Factor Design 
— Both Factors Repeated 



Source of Variation 



d.f . 



Sum of 
Squares 



Mean 
Square 



Error 

Term 



Factors 



19.26 



3.21 



3.75 



(Fx J) 



System 
Judges 

Factors x Systems 



24 



Factors x Judges (FxJ) 48 



Systems x Judges (SxJ) 32 



27.00 
31.53 
17.53 
41.14 

31.17 



6.75 



3.94 



6.93 



(SxJ) 



•73 1.55 (FxSxJ) 



.857 



.974 



Factors x Systems 

X Judges (FxSxJ) 



192 



93.50 



.47 



TOTAL 



314 
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p < .01 
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The development of a judgmental criterion expands our arsenal of 
tools for program evaluation. It provides an additional data-point 
to use in real world program evaluations. This 'Wltiple data-point 
method" permits us to characterize the delivery systems on several 
different criteria. Several of these characteristics are shown in 
Table 4. 

[Insert Table 4 here] 
Cost per teacher participation hour is computed by dividing the 
cost of a project by the total number of man-hours of instruction 
provided. It is a straightforward calculation, although budget figures 
are considerably raiore difficult to obtain. The figures in Table 4 are 
based on expenditure estimates with an accuracy of - 5%. 

Use rates — that is, the percent of teachers using the innovation 
the year following the training — are more variable. In some instances, 
it is a new course, e.g., BSCS biology, while in others it may be a 
single instructional unit, e.g., a rock identification lab. 

Satisfaction of teachers with the training is in response to a 
questionnaire given to participants following the workshop. Nearly all 
(94%) teachers expressed a positive response to questions dealing with 
the general effectiveness of the delivery systems. Although the magni- 
tude of these figures should be viewed with caution because of unknox^ 
bias in the responses, teachers apparently find this program useful. 

Although five cases do not provide many data points, there appears 
to be no strong correlation among the variables. For example, spending 
more money does not necessarily insure increased use rates. Each criterion 
seems to be a different way of evaluating the delivery system. 



System 



Accessible 

Centers 

Collaborative 

Portal 

Workshop 



Table 4 

Delivery System Characteristics 



D&U Factor 
Ratina 

2.87 

3.02 

3.43 

3.54 

2.82 



NSF 
Cost Per 
Teacher 
Participa- 
tion Hour 



$ 9.50 
$17.10 
$ 7.80 
$ 1.50 
$ 3.20 



Mean 



3.14 



$ 7.80 



Criteria 



First Year 
Use Rates 



84Z 
67% 
96% 
48% 
51% 



Participant 
Satisfaction 
Rating* 



100% 



90% 
90% 
98% 
93% 



94% 



Percent indicating 
positive reaction 
to delivery system 
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Perhaps as more projects are evaluated using these procedures, some 
trends will emerge. Meanwhile, it is important to note that variations 
do occur and that the mean figures best characterize this implementa- 
tion program* Participants were very satisfied, judges rated the 
program about average, cost ia $7.80 per TPH, and first year adoption 
rate is around 70 percent. 

0 

V. Discussion 

A major purpose of this study was to develop additional criteria that 
could be used to evaluate NSF sponsored implementation projects. We have 
shown that discrimination among various types is possible using judges and 
the Havelock factors. Further study is needed to examine the relationships 
between this quality criteria and other evaluative criteria — for example, 
adoption rates, cost, persistence of implementation, and participant satis- 
faction. Some evidence has been provided (Table 4) which indicates the 
existence of independent criteria. Because of this possibility, it is our 
belief that a multiplicity of variables should be used in program evaluations. 

This study has also highlighted several characteristics of the Founda- 
tion's implementation program. > The projects vary considerably on training 
costs (range $1.40 to $17.00) and on use or adoption rates (51% to 96%). The 
projects are conducted near the teachers (proximity) , carried out by competent 
staff (capacity), and organized to fit users' adoption needs (structure). 
On the other hand, several shortcomings of the program emerge. NSF supported 
projects of this type are usually single mode (low synergy), and do not 
permit two-way interaction between resource and user (low openness). 
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Proje^nt directors and the National Science Foundation need to consider 
these strengths and limitations as they seek ways to increase program effec- 
tiveness. Hopefully, the information generated in this and other evaluation 
studies can serve accountability needs of NSF and provide a basis for 
improved decision making. 
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